Add support for Amazon OpenSearch Serverless by lawofcycles · Pull Request #586 · opensearch-project/opensearch-hadoop

lawofcycles · 2025-05-17T13:04:54Z

Description

Add support for Amazon OpenSearch Serverless.

When opensearch.serverless=true is set, opensearch-hadoop works in serverless mode, connecting to serverless endpoints and calling APIs based on serverless specs.

How it works

Write path: Adapted to Serverless APIs. Cluster health checks, refresh calls, and node discovery are skipped since these endpoints are not available on Serverless. ClusterInfo is synthesized locally without calling GET /.

Read path: Uses PIT (Point in Time) + Search After API for pagination. Scroll API and Slice API are not available on Serverless, so all reads go through a single partition per index. PIT ensures snapshot consistency during reads, and _id is used as a sort tiebreaker to prevent missing documents.

Serverless API	Status	Notes
`POST /_search/scroll`	Not supported	Returns 404
`POST /<index>/_search/point_in_time`	Supported	Used for consistent reads
`slice` parameter	Not supported	Returns 500
`search_after` + `pit`	Supported	Primary read pagination method

Configuration

Setting	Default	Description
`opensearch.serverless`	`false`	Enable serverless mode
`opensearch.pit.keepalive`	`5m`	PIT keep alive duration for serverless reads
`opensearch.nodes.wan.only`	Must be `true`	Required for serverless

Example

Write

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.createDataFrame([
  (1, "text1"),
  (2, "text2")
], ["id", "text"])

df.write \
  .format("org.opensearch.spark.sql") \
  .option("opensearch.nodes", "<opensearch serverless endpoint>") \
  .option("opensearch.port", "443") \
  .option("opensearch.resource", "<index name>") \
  .option("opensearch.aws.sigv4.enabled", "true") \
  .option("opensearch.aws.sigv4.region", "<region name>") \
  .option("opensearch.nodes.wan.only", "true") \
  .option("opensearch.serverless", "true") \
  .mode("append") \
  .save()

Read

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.read \
  .format("org.opensearch.spark.sql") \
  .option("opensearch.nodes", "<opensearch serverless endpoint>") \
  .option("opensearch.port", "443") \
  .option("opensearch.resource", "<index name>") \
  .option("opensearch.aws.sigv4.enabled", "true") \
  .option("opensearch.aws.sigv4.region", "<region name>") \
  .option("opensearch.nodes.wan.only", "true") \
  .option("opensearch.serverless", "true") \
  .load()

df.show()

Future consideration

The current Search After implementation reuses ScrollReader and ScrollQuery with a serverless mode flag. Ideally, the pagination logic (ScrollQuery) should be split into separate ScrollPaginationQuery and SearchAfterPaginationQuery implementations, with the common hit parsing logic in ScrollReader extracted into a shared base. However, ScrollReader.read() contains complex parsing, raw JSON reconstruction, and metadata extraction that accounts for the majority of the code, while the Scroll/SearchAfter divergence is limited to response header handling and a few lines of state management. Restructuring this carries regression risk disproportionate to the benefit, so this refactoring is left for a future PR.

Testing

Tested on EMR 7.12 with Spark, Hive, and MapReduce against OpenSearch Serverless. Also verified that the standard (non serverless) read/write path is unaffected.

Co-authored-by: @dimorportheca47

Issues Resolved

fix #269

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

lawofcycles · 2025-05-29T22:56:08Z

I have conducted the compatibility test for all OpenSearch APIs called from OpenSearch Hadoop.
Some of them are not supported on Serverless and we need further implementation to fill those gaps. I'm working on them.

API	Status	Implementation
Cluster Management
`GET /`	Not supported -> Handled	Returns dummy ClusterInfo (name="serverless-collection", version=V_2_X)
`GET /_nodes/http`	Not supported -> Skipped	Disabled via `opensearch.nodes.discovery=false`
`GET /_cluster/health/{index}`	Not supported -> Skipped	Returns immediately in serverless mode
Search
`GET/POST /{index}/_search`	Supported	Works as-is
`POST /_search/scroll`	Not supported (404)	Needs Search After API as replacement (see below)
`DELETE /_search/scroll`	Not supported -> Ignored	Scroll context expires automatically via timeout
`GET /{index}/_search_shards`	Not supported -> Handled	Creates single partition per index via `findServerlessPartitions()`
`POST /{index}/_search` with `slice`	Not supported (500)	No parallel read support on Serverless
`POST /{index}/_search` with `search_after`	Supported	Proposed replacement for Scroll API
`POST /{index}/_count`	Supported	Works as-is
Index Management
`HEAD/PUT/DELETE /{index}`	Supported	Works as-is
`GET/PUT /{index}/_mapping`	Supported	Works as-is
`POST /{index}/_refresh`	Not supported -> Skipped	No-op in serverless (auto-refresh)
`GET /{index}/_aliases`	Supported	Works as-is
Document
`POST /_bulk`	Supported	Works as-is
`POST /{index}/_doc`	Supported	Works as-is
`POST /{index}/_delete_by_query`	Not supported -> Fallback	Skipped, falls back to scroll + bulk delete

20260216Update

After further testing, I found that POST /_search/scroll returns 404 Not Found on OpenSearch Serverless. The initial search with ?scroll=1m succeeds and returns a _scroll_id, but the continuation endpoint does not exist. This means the current read implementation cannot paginate beyond the first batch (opensearch.scroll.size, default 1,000 docs). Note that opensearch.scroll.size can be increased up to 10,000, but that is the hard limit on Serverless since index.max_result_window is fixed at 10,000 and cannot be modified.

The slice parameter returns 500 Internal Server Error, so parallel partition reads via opensearch.input.max.docs.per.partition are not possible on Serverless. All reads go through a single partition.

The only viable pagination method on Serverless is PIT + Search After API (pit + sort + search_after), which provides consistent snapshot reads. The sort should include _id as a tiebreaker ("sort": ["_doc", "_id"]) to prevent missing documents caused by duplicate _doc sort values on Serverless.

achaiban · 2025-06-19T07:46:10Z

Thanks @lawofcycles for your work on this so far. I see you seem to have patched the sink related API request handling for AOSS mentioned here: #269 (comment)

If you could please provide me a build I'd love to test df sink with Glue 4.0 and report back, Gradle is new to me so I haven't been able to build the patch myself.

lawofcycles · 2025-07-13T07:15:17Z

@achaiban Thank you for your suggestion and sorry for the late response.
I was a bit busy recently so I couldn't work on this, but I'm back on it now.
Once my work reaches a testable stage, I'll be pleased to share it with you. Please bear with me a little longer.
Currently, I'm testing my patch on Spark on EMR.

lawofcycles · 2025-12-23T05:34:53Z

@Xtansia @harishbhakuni @nknize
@dimorportheca47 and I have finally developed serverless support. We tested it with Spark, Hive, and MapReduce on EMR 7.12.
Could you take a look?

Co-authored-by: fukamishuhei <dimorportheca.0407@gmail.com> Signed-off-by: Sotaro Hikita <bering1814@gmail.com>

…rless mode Signed-off-by: Sotaro Hikita <bering1814@gmail.com>

lawofcycles · 2026-02-15T03:46:33Z

I pushed a new commit to replace Scroll API with Search After API for read pagination in serverless mode.

POST /_search/scroll returns 404 on Serverless, so the read path was broken for indices larger than opensearch.scroll.size (default 1,000 docs). The new commit uses the Search After API (sort + search_after) to paginate through all documents.

Tested on EMR 7.12 with a 100,000 doc index on OpenSearch Serverless. Both read and write operations work correctly. Also verified that the standard (non serverless) read/write path is unaffected.

Future consideration

The current Search After implementation reuses ScrollReader and ScrollQuery with a serverless mode flag. Ideally, the pagination logic (ScrollQuery) should be split into separate ScrollPaginationQuery and SearchAfterPaginationQuery implementations, with the common hit parsing logic in ScrollReader extracted into a shared base. However, ScrollReader.read() contains complex parsing, raw JSON reconstruction, and metadata extraction that accounts for the majority of the code, while the Scroll/SearchAfter divergence is limited to response header handling and a few lines of state management. Restructuring this carries regression risk disproportionate to the benefit, so this refactoring is left for a future PR.

…null for serverless compatibility Signed-off-by: Sotaro Hikita <bering1814@gmail.com>

…issing documents in serverless search_after reads Signed-off-by: Sotaro Hikita <bering1814@gmail.com>

lawofcycles · 2026-02-16T06:00:56Z

I pushed another commit to add PIT (Point in Time) support and a _id sort tiebreaker for serverless reads.

The previous Search After implementation had two issues.

Missing documents: sort: ["_doc"] alone produces duplicate sort values on Serverless (all docs in a batch share the same value), causing search_after to skip unread documents at batch boundaries. Adding _id as a tiebreaker (sort: ["_doc", "_id"]) ensures unique sort values and eliminates the data loss.
No snapshot consistency: Search After without PIT is stateless, so concurrent writes or segment merges during reads could cause duplicates or gaps. Each read session now creates a PIT at the start and deletes it on close, ensuring all pages are read from a consistent snapshot.

Tested on EMR 7.12 with OpenSearch Serverless and provisioned OpenSearch.

lawofcycles · 2026-02-16T08:34:43Z

@Xtansia @nknize @harshavamsi I've updated the PR description to reflect the latest changes. Ready for review.

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi · 2026-02-19T15:05:42Z

+        if (this.settings.getServerlessMode()) {
+            // Use a dummy UUID instead of null to avoid NPE in validation
+            ClusterName clusterName = new ClusterName("serverless-collection", "serverless-uuid");
+            return new ClusterInfo(clusterName, OpenSearchMajorVersion.V_2_X);


So we should definitely upgrade the client to support OS versions 3 and above, but it will be a breaking change that we can address later

Good catch, I'll raise PR later.

After looking into this further, the version branching in the codebase is mostly limited to the type/typeless API distinction (V_1_X vs V_2_X+), and V_3_X is already defined and parseable in OpenSearchMajorVersion. Since Serverless does not expose its internal OpenSearch version through GET /, it is unclear what version should be returned here.
The current V_2_X works correctly because all existing version branches treat V_2_X and above identically (typeless path). I think we can revisit this when a concrete incompatibility surfaces, either from Serverless behavior changes.

harshavamsi · 2026-02-19T15:07:34Z

-        // try first a blind delete by query
-        try {
-            Resource res = resources.getResourceWrite();
-            client.deleteByQuery(
-                    res.isTyped()
-                            ? res.index() + "/" + res.type()
-                            : res.index(),
-                    MatchAllQueryBuilder.MATCH_ALL);
-        } catch (OpenSearchHadoopInvalidRequest ehir) {
-            log.error("Delete by query was not successful...", ehir);


does serverless not support delete?

nvm, saw the description

lawofcycles · 2026-02-20T19:27:02Z

@harshavamsi
I've been testing various write/read patterns against OpenSearch Serverless and found a couple of issues. Submitted follow-up PRs:

Critical fixes:

Skip refreshIndex in serverless mode to fix 404 on _refresh endpoint #692: Fix serverless mode writes failing with 404 on _refresh endpoint ([BUG] Serverless mode: writes fail with 404 on refresh endpoint #690)
Use PIT + search_after for delete in serverless mode (#691) #693: Fix SaveMode.Overwrite failing when document count exceeds scroll size in serverless mode ([BUG] Serverless mode: SaveMode.Overwrite fails when document count exceeds scroll size #691)

Minor improvement:

Add opensearch.search_after.size setting for serverless mode page size #695: Add dedicated opensearch.search_after.size setting for serverless mode page size ([FEATURE] Add dedicated page size setting for serverless mode (search_after pagination) #694)

All verified on EMR 7.12 + OpenSearch Serverless. Would appreciate a review when you get a chance.

advaitpathak-cod · 2026-03-12T16:42:30Z

Hi @harshavamsi,

Thanks a lot for adding support for OpenSearch Serverless - this is really appreciated. I just wanted to check when this change is expected to be included in a Maven release and which version it will be part of.

At the moment, the latest version available on Maven Central is 1.3.0:
https://mvnrepository.com/artifact/org.opensearch.client/opensearch-spark-30_2.12/1.3.0

It would be helpful to know when a new release including this PR might be published.

lawofcycles · 2026-03-13T08:26:05Z

@advaitpathak-cod
Hi, I'm planning the new release for opensearch-hadoop.
There is no fixed timeline yet, but I'm currently working on confirming the tasks and schedule.

#685
#699

If you need a version with serverless support before that, I can show you how to build the jar.

advaitpathak-cod · 2026-03-13T11:05:46Z

@advaitpathak-cod Hi, I'm planning the new release for opensearch-hadoop. There is no fixed timeline yet, but I'm currently working on confirming the tasks and schedule.

#685 #699

If you need a version with serverless support before that, I can show you how to build the jar.

Thanks @lawofcycles for the update.
I can wait for the serverless support as part of the regular release.

lawofcycles force-pushed the add-support-for-opensearch-serverless branch 3 times, most recently from 14e2743 to 8abad24 Compare May 18, 2025 00:48

lawofcycles force-pushed the add-support-for-opensearch-serverless branch from 4423c97 to c1987f9 Compare July 13, 2025 10:51

lawofcycles force-pushed the add-support-for-opensearch-serverless branch 2 times, most recently from 7636681 to 65c28df Compare December 23, 2025 04:39

lawofcycles changed the title ~~WIP: Add support for Amazon OpenSearch Serverless~~ Add support for Amazon OpenSearch Serverless Dec 23, 2025

lawofcycles marked this pull request as ready for review December 23, 2025 04:40

lawofcycles requested review from Xtansia, harshavamsi and nknize as code owners December 23, 2025 04:40

lawofcycles force-pushed the add-support-for-opensearch-serverless branch from 65c28df to e4a7fb8 Compare December 23, 2025 05:01

lawofcycles force-pushed the add-support-for-opensearch-serverless branch from bda9f75 to daacaee Compare December 23, 2025 05:48

lawofcycles and others added 2 commits February 15, 2026 09:58

Add support for Amazon OpenSearch Serverless

aaccda3

Co-authored-by: fukamishuhei <dimorportheca.0407@gmail.com> Signed-off-by: Sotaro Hikita <bering1814@gmail.com>

Replace Scroll API with Search After API for read pagination in serve…

b5e1963

…rless mode Signed-off-by: Sotaro Hikita <bering1814@gmail.com>

lawofcycles force-pushed the add-support-for-opensearch-serverless branch from daacaee to b5e1963 Compare February 15, 2026 03:44

Fix testNoScrollIdFromFrozenIndex to expect Scroll object instead of …

5473f02

…null for serverless compatibility Signed-off-by: Sotaro Hikita <bering1814@gmail.com>

lawofcycles force-pushed the add-support-for-opensearch-serverless branch from 33cf2e3 to 5473f02 Compare February 15, 2026 06:58

Add PIT support and _id sort tiebreaker to fix data consistency and m…

497179e

…issing documents in serverless search_after reads Signed-off-by: Sotaro Hikita <bering1814@gmail.com>

Merge branch 'main' into add-support-for-opensearch-serverless

354f188

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

harshavamsi reviewed Feb 19, 2026

View reviewed changes

harshavamsi approved these changes Feb 19, 2026

View reviewed changes

harshavamsi merged commit 9807b25 into opensearch-project:main Feb 19, 2026
16 checks passed

Uh oh!

Conversation

lawofcycles commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How it works

Configuration

Example

Write

Read

Future consideration

Testing

Issues Resolved

Uh oh!

lawofcycles commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

20260216Update

Uh oh!

achaiban commented Jun 19, 2025

Uh oh!

lawofcycles commented Jul 13, 2025

Uh oh!

lawofcycles commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lawofcycles commented Feb 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Future consideration

Uh oh!

lawofcycles commented Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lawofcycles commented Feb 16, 2026

Uh oh!

harshavamsi Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

lawofcycles Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

lawofcycles Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshavamsi Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

harshavamsi Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lawofcycles commented Feb 20, 2026

Uh oh!

advaitpathak-cod commented Mar 12, 2026

Uh oh!

lawofcycles commented Mar 13, 2026

Uh oh!

advaitpathak-cod commented Mar 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lawofcycles commented May 17, 2025 •

edited

Loading

lawofcycles commented May 29, 2025 •

edited

Loading

lawofcycles commented Dec 23, 2025 •

edited

Loading

lawofcycles commented Feb 15, 2026 •

edited

Loading

lawofcycles commented Feb 16, 2026 •

edited

Loading

lawofcycles Feb 23, 2026 •

edited

Loading